Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning

The widespread adoption of commercial autonomous vehicles (AVs) and advanced driver assistance systems (ADAS) may largely depend on their acceptance by society, for which their perceived trustworthiness and interpretability to riders are crucial. In general, this task is challenging because modern autonomous systems software relies heavily on black-box artificial intelligence models. Towards this goal, this paper introduces a novel dataset, Rank2Tell, a multi-modal ego-centric dataset for Ranking the importance level and Telling the reason for the importance. Using various close and open-ended visual question answering, the dataset provides dense annotations of various semantic, spatial, temporal, and relational attributes of various important objects in complex traffic scenarios. The dense annotations and unique attributes of the dataset make it a valuable resource for researchers working on visual scene understanding and related fields. Further, we introduce a joint model for joint importance level ranking and natural language captions generation to benchmark our dataset and demonstrate performance with quantitative evaluations

AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing what commonly happens after his/her current action (e.g. crack eggs)? What if we also know the longer-term goal of the actor (e.g. making egg fried rice)? The long-term action anticipation (LTA) task aims to predict an actor's future behavior from video observations in the form of verb and noun sequences, and it is crucial for human-machine interaction. We propose to formulate the LTA task from two perspectives: a bottom-up approach that predicts the next actions autoregressively by modeling temporal dynamics; and a top-down approach that infers the goal of the actor and plans the needed procedure to accomplish the goal. We hypothesize that large language models (LLMs), which have been pretrained on procedure text data (e.g. recipes, how-tos), have the potential to help LTA from both perspectives. It can help provide the prior knowledge on the possible next actions, and infer the goal given the observed part of a procedure, respectively. To leverage the LLMs, we propose a two-stage framework, AntGPT. It first recognizes the actions already performed in the observed videos and then asks an LLM to predict the future actions via conditioned generation, or to infer the goal and plan the whole procedure by chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2 benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the effectiveness of our proposed approach. AntGPT achieves state-of-the-art performance on all above benchmarks, and can successfully infer the goal and thus perform goal-conditioned "counterfactual" prediction via qualitative analysis. Code and model will be released at https://brown-palm.github.io/AntGP

Assessment of left ventricular systolic and diastolic function in juvenile rheumatoid arthritis

Background and Aims: Recognizing the paucity of data regarding echocardiographic studies of Left ventricular (LV) systolic and diastolic function in patients with juvenile rheumatoid arthritis (JRA), a study was carried out to study these parameters in these subjects. Settings, Design and Methods: Thirty-five patients with JRA and an equal number of age- and sex-matched controls were studied by two-dimensional and Doppler echocardiography. Results: Patients with JRA had higher systolic and diastolic blood pressures, resting heart rates, LV systolic (26.9\ub14.3 vs. 22.4 \ub1 4.1 mm, p=0.001) and diastolic size (42.3\ub14.6 vs. 35.4\ub13.8 mm, p<0.001) and volumes. Though ejection fraction (EF) and fractional shortening (FS) were normal, they were lower in those with JRA as compared to controls (EF: 62.9\ub14.47 vs. 67.5\ub13.63 %, p<0.001; FS: 36.4\ub14.5 vs. 38.5 \ub1 6.87, p=0.2). On Doppler analysis the JRA group had lower peak E velocity, higher peak A velocity, higher A VTI and more prolonged IVRT. Male patients had higher A VTI and IVRT as compared to females. Those with longer duration of disease had larger LV systolic (r=0.517, p=0.01) and diastolic dimension (r=0.40, p=0.05) and lower FS (r=-0.506, p=0.01). Patients with polyarticular JRA had higher E and A VTI as compared to those with systemic or oligoarticular types. Conclusion: Despite an asymptomatic cardiac status, significant systolic and diastolic functional abnormalities exist in patients with JRA. The duration of the disease, mode of presentation, patient's age and gender have a significant impact on the left ventricular systolic and diastolic functions in patients with JRA

Learning to Recognise Objects and Actions for Intelligent Agents

Computer vision involves a host of tasks, such as boundary detection, semantic segmentation, surface estimation, object detection, image classification, action localization, to name a few. For a holistic understanding of a scene, which is required by a lot of real-world applications, many of these tasks need to be combined together. For instance, an autonomous car should not only be able to detect other cars (object) but also if a pedestrian is walking (action). The former requires localizing the object, which can either be at the pixel level or bounding box level. The latter requires localizing the action, and by extension the actor, in both space and time. These problems are best dealt with approaches involving supervised learning models which rely on large annotated datasets, and so the problem becomes even more challenging when there is lack of labeled data.In this thesis, we first tackle the problem of spatio-temporal action localization in an unsupervised setting. As the name suggests, it requires modeling of both spatial and temporal features. So, we propose an end-to-end learning framework for an adaptation method which aligns both spatial and temporal features and conduct experiments on the action localization task. To highlight the potential benefits for autonomous cars, we also construct and benchmark a new dataset which contains pedestrian actions collected in driving scenes. Then, for a holistic understanding of the scene, we shift our attention from localizing actions to recognising objects especially in a city street scenario. We do this by jointly dealing with the tasks of object detection and semantic segmentation. While the former localizes the individual instances of objects at the bounding box level, the latter provides pixel level distinction but at the category level. We explore a novel observation that connects the two tasks and provide an end-to-end learning framework to exploit this connection

Improving multiclass classification by deep networks using DAGSVM and Triplet Loss

With recent advances in the field of computer vision and especially deep learning, many fully connected and convolutional neural networks have been trained to achieve state-of-the-art performance on a wide variety of tasks such as speech recognition, image classification and natural language processing. For classification tasks however, most of these deep learning models employ the softmax activation function for prediction and minimize cross-entropy loss. In contrast, we demonstrate a consistent advantage by replacing the softmax layer by a set of binary SVM classifiers organized in a tree or DAG (Directed Acyclic Graph) structure. The idea is to not treat the multiclass classification problem as a whole but to break it down into smaller binary problems where each classifier acts as an expert by focusing on differentiating between only two classes and thus improves the overall accuracy. Furthermore, by arranging the classifiers in a DAG structure, we later also show how it is possible to further improve the performance of the binary classifiers by learning more discriminative features through the same deep network. We validated the proposed methodology on two benchmark datasets, and the results corroborated our claim

Assessment of left ventricular systolic and diastolic function in juvenile rheumatoid arthritis

Background and Aims: Recognizing the paucity of data regarding echocardiographic studies of Left ventricular (LV) systolic and diastolic function in patients with juvenile rheumatoid arthritis (JRA), a study was carried out to study these parameters in these subjects. Settings, Design and Methods: Thirty-five patients with JRA and an equal number of age- and sex-matched controls were studied by two-dimensional and Doppler echocardiography. Results: Patients with JRA had higher systolic and diastolic blood pressures, resting heart rates, LV systolic (26.9&#x00B1;4.3 vs. 22.4 &#x00B1; 4.1 mm, p=0.001) and diastolic size (42.3&#x00B1;4.6 vs. 35.4&#x00B1;3.8 mm, p<0.001) and volumes. Though ejection fraction (EF) and fractional shortening (FS) were normal, they were lower in those with JRA as compared to controls (EF: 62.9&#x00B1;4.47 vs. 67.5&#x00B1;3.63 &#x0025;, p<0.001; FS: 36.4&#x00B1;4.5 vs. 38.5 &#x00B1; 6.87, p=0.2). On Doppler analysis the JRA group had lower peak E velocity, higher peak A velocity, higher A VTI and more prolonged IVRT. Male patients had higher A VTI and IVRT as compared to females. Those with longer duration of disease had larger LV systolic (r=0.517, p=0.01) and diastolic dimension (r=0.40, p=0.05) and lower FS (r=-0.506, p=0.01). Patients with polyarticular JRA had higher E and A VTI as compared to those with systemic or oligoarticular types. Conclusion: Despite an asymptomatic cardiac status, significant systolic and diastolic functional abnormalities exist in patients with JRA. The duration of the disease, mode of presentation, patient&#x0027;s age and gender have a significant impact on the left ventricular systolic and diastolic functions in patients with JRA

Angiotensin-converting enzyme gene polymorphism in coronary artery disease in north India

Background: The aim of this study was to investigate the role of angiotensin-converting enzyme gene polymorphism in patients with coronary artery disease in north India. Methods and Results: One hundred forty-six patients with angiographically proven atherosclerotic coronary artery disease, and 146 age- and sex-matched control subjects (treadmill-negative) were included in the study. Genomic DNA was extracted and analyzed for angiotensin-converting enzyme insertion/deletion polymorphism. Two independent investigators scored the genotypes. Conclusions: When we compared the genotypes of patients with coronary artery disease with those of normal controls, it was seen that all three genotypes, i.e. DD, ID and II, were not statistically different among patients and controls. Further, we categorized the patient and control groups into 2 subgroups, i.e. below and above 50 years of age. Interestingly, it was observed that the DD genotype was significantly higher in patients in the higher age group (i.e. above 50 years of age). However, this needs further validation by studying patients with coronary artery disease from other parts of India

Angiotensin-converting enzyme gene polymorphism in coronary artery disease in north India

Background: The aim of this study was to investigate the role of angiotensin-converting enzyme gene polymorphism in patients with coronary artery disease in north India. Methods and Results: One hundred forty-six patients with angiographically proven atherosclerotic coronary artery disease, and 146 age- and sex-matched control subjects (treadmill-negative) were included in the study. Genomic DNA was extracted and analyzed for angiotensin-converting enzyme insertion/deletion polymorphism. Two independent investigators scored the genotypes. Conclusions: When we compared the genotypes of patients with coronary artery disease with those of normal controls, it was seen that all three genotypes, i.e. DD, ID and II, were not statistically different among patients and controls. Further, we categorized the patient and control groups into 2 subgroups, i.e. below and above 50 years of age. Interestingly, it was observed that the DD genotype was significantly higher in patients in the higher age group (i.e. above 50 years of age). However, this needs further validation by studying patients with coronary artery disease from other parts of India